Bayesian controller fusion: Leveraging control priors in deep reinforcement learning for robotics

نویسندگان

چکیده

We present Bayesian Controller Fusion (BCF): a hybrid control strategy that combines the strengths of traditional hand-crafted controllers and model-free deep reinforcement learning (RL). BCF thrives in robotics domain, where reliable but suboptimal priors exist for many tasks, RL from scratch remains unsafe data-inefficient. By fusing uncertainty-aware distributional outputs each system, arbitrates between them, exploiting their respective strengths. study on two real-world tasks involving navigation vast long-horizon environment, complex reaching task involves manipulability maximisation. For both these domains, simple handcrafted can solve at hand risk-averse manner do not necessarily exhibit optimal solution given limitations analytical modelling, controller miscalibration variation. As exploration is naturally guided by prior early stages training, accelerates learning, while substantially improving beyond performance prior, as policy gains more experience. More importantly, risk-aversity ensures safe deployment, dominates action distribution states unknown to policy. additionally show BCF’s applicability zero-shot sim-to-real setting its ability deal with out-of-distribution real world. promising approach towards combining complementary robotic control, surpassing what either achieve independently. The code supplementary video material are made publicly available https://krishanrana.github.io/bcf .

برای دانلود باید عضویت طلایی داشته باشید

برای دانلود متن کامل این مقاله و بیش از 32 میلیون مقاله دیگر ابتدا ثبت نام کنید

اگر عضو سایت هستید لطفا وارد حساب کاربری خود شوید

منابع مشابه

Nonparametric Bayesian Policy Priors for Reinforcement Learning

We consider reinforcement learning in partially observable domains where the agent can query an expert for demonstrations. Our nonparametric Bayesian approach combines model knowledge, inferred from expert information and independent exploration, with policy knowledge inferred from expert trajectories. We introduce priors that bias the agent towards models with both simple representations and s...

متن کامل

PAC-MDP Reinforcement Learning with Bayesian Priors

In an effort to build on recent advances in reinforcement learning and Bayesian modeling, this work (Asmuth et al., 2009) combines ideas from two lines of research on exploration in reinforcement learning or RL (Sutton & Barto, 1998). Bayesian RL research (Dearden et al., 1999; Poupart et al., 2006) formulates the RL problem as decision making in the belief space of all possible environment mod...

متن کامل

Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards

We propose a general and model-free approach for Reinforcement Learning (RL) on real robotics with sparse rewards. We build upon the Deep Deterministic Policy Gradient (DDPG) algorithm to use demonstrations. Both demonstrations and actual interactions are used to fill a replay buffer and the sampling ratio between demonstrations and transitions is automatically tuned via a prioritized replay me...

متن کامل

Dual Control for Approximate Bayesian Reinforcement Learning

Control of non-episodic, finite-horizon dynamical systems with uncertain dynamics poses a tough and elementary case of the exploration-exploitation trade-off. Bayesian reinforcement learning, reasoning about the effect of actions and future observations, offers a principled solution, but is intractable. We review, then extend an old approximate approach from control theory—where the problem is ...

متن کامل

Benchmarking Deep Reinforcement Learning for Continuous Control

Recently, researchers have made significant progress combining the advances in deep learning for learning feature representations with reinforcement learning. Some notable examples include training agents to play Atari games based on raw pixel data and to acquire advanced manipulation skills using raw sensory inputs. However, it has been difficult to quantify progress in the domain of continuou...

متن کامل

ذخیره در منابع من


  با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید

ژورنال

عنوان ژورنال: The International Journal of Robotics Research

سال: 2023

ISSN: ['1741-3176', '0278-3649']

DOI: https://doi.org/10.1177/02783649231167210